COMMENT ⊗   VALID 00002 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.require "memo.pub[let,jmc]" source
C00023 ENDMK
C⊗;
.require "memo.pub[let,jmc]" source

.once center
%3Notes on Searle's "Notes on artificial intelligence"%1

	I think that suitable computer programs can %2understand%1.
However, such a program is not a theory in itself any more than a man who
understands is a theory.  Admittedly there is a bobtailed theory of
understanding based on any program or man who understands.  The theory is
%2understanding is the internal structure required to account for the
behavior of this program or man%1.  The reader is invited to read the
listing of the program or study the physiology of the man.  Such theories
of understanding won't be very illuminating unless the program is more
comprehensible than any of today's substantial programs or unless
physiology becomes much easier than it has been - both technically and
conceptually.

	It would be more reasonable to regard a program that is claimed to
understand as an illustration of a theory of understanding and an
existence proof that the form of %2understanding%1 built into the program
meets some of our intuitive desiderata or at least meets the empirical
claims of its designer.  To serve as such an illustration, the program
must be accompanied by an explanation of how it embodies the concept of
understanding being illustrated.

	It is conceivable that someone will make a program that
understands without being able to explain it satisfactorily.  I
think that is unlikely and would be very disappointed should it
turn out that by tinkering, humanity can produce artificial
intelligence but cannot understand intelligence even with the
aid of the intelligent programs themselves.

	Therefore, it would be better for me if Searle would split
his stronger claim about AI into two versions - one that says
programs can potentially understand and the other that such a
program would constitute a theory of understanding all by itself.

1a I have reservations about Schank's program and also about
Schank's doctrines about what constitutes a theory of AI.

	I am doubtful that Schank's program understands, but I don't
have a formal definition of understanding with which to compare it.
I am not convinced by the dialogs cited, and I think I could invent
other questions using the same vocabulary that would buffalo the
program.  I don't know if the program would be able to answer whether
the pleased man would have accepted an offer from the waitress to eat the
hamburger again or whether the offended man would have accepted an
offer to uncook the hamburger.

	I am also uncertain whether
these questions are really criterial for understanding about restaurants.
What if the program were asked, "Might the offended man have decided
that he would never eat again".  I choose these outlandish questions,
because I suspect that the program, like many AI programs, is fragile.
It seems to make sense if you respect it, but if you are skeptical
enough, you can elicit ridiculous behavior.  Unfortunately, this
off-the-cuff skepticism is not based on having read enough of Schank's
papers well enough to have a fully informed opinion.  It would be
worthwhile to have the opinion of someone who has read more, and,
if doubt remains, to perform the experiment.
(I remember a ⊗go program that made rather good moves if you made
reasonable opening moves, but collapsed if you launched unsound
direct attacks on its groups.  This collapse contrasts with the
behavior of even the microcomputer chess games that will punish
sufficiently unsound play without regard to any expectations of
sound play).

	There is a level of performance that would leave me betting
that Schank's program really understands, but without more of a
theory of understanding, I wouldn't easily be finally convinced.

	I like the "Berkeley answer" that the system knows and will defend
it against Searle's objections.

1.  Suppose that the human doesn't know Chinese, only the set of rules for
making responses to the strings of Chinese characters.  Suppose further that
the rules are elaborate enough to behave like an excellent Chinese scholar.

	I would then say that this system, though not the "underlying
mental system" of the person understands Chinese.  That a person could
carry out rules elaborate enough to behave like a Chinese scholar is
implausible given the limitations of human data processing capability.
Moreover, AI is not ready to formulate the required rules.

2.  There are some weakly analogous cases, however. Some schizophrenics are said
to have split personalities - the same brain carries out the mental processes
of the several personalities.

3.  Searle's problem can be mapped into a purely computer framework.  Suppose
a program, like the proposed Advice Taker, formulates what it knows as
sentences of logic and decides what to do by reasoning in this logic.  Suppose
further that this program can engage in reasonably intelligent dialog - in
the logical language or even in English.  Suppose further that someone gives
the program - in logic or in English - a set of rules for manipulating
pictographs but gives no rules for translating the pictographs into its
basic logical language.  Suppose further that the rules again amount to
the behavior of a Chinese scholar.  Then the interpreted system knows
Chinese and many facts about Chinese literature while the basic system
doesn't.

Actually, the same problems would arise if the interpreted language were English,
except that the basic system would have to be rather obtuse to obey the
rules given in English for manipulating quoted strings of characters without
noticing that these strings could themselves be interpreted in its base language.

Searle's paper points up the fact that a given brain could be host to several
minds.  These minds could either be parallel in the brain in that the %2hardware%1
interprets them separately or they can be organized in layers - where
one mind knows how to obey the rules of another with or without being able to 
translate the other's sentences into its own language.

If such a phenomenon were common, ordinary language would not make the "category 
mistake" of saying "John saw himself in the mirror", since it would not
identify the personality John with the physical object.  Since the phenomenon
probably doesn't actually occur except with computers, ordinary language
users make this mistake only in connection with computers and
officialdoms.  The statement "The computer knows X" or "The government
knows X" often elicits the reply "What program are you talking about, do
you mean the personnel program that has just been moved from an IBM
370/168 to an IBM 3033?" or "What agency do you mean?".

	I turn now to the interpolation on understanding.  In my paper
%2Ascribing Mental Qualities to Machines%1, I advocate ascribing limited
beliefs even to thermostats, even though it doesn't contribute to
understanding them.  (It might for a person with a sufficiently limited
knowledge faced with a slightly complex thermostatic system).

	Some of the arguments against such ascription resemble proposals to
begin the number system with 2 on the grounds that if we only had to
consider sets with 0 and 1 elements, we wouldn't have any real use for
numbers.  The answer is that starting with 2 would give the general laws
dealing with numbers a less comprehensible form.  Indeed for a long time 0
was mistakenly omitted from the number system on the grounds that it
wasn't really a number.  My intuition is that a good theory of belief will
also admit trivial cases that wouldn't in themselves justify the theory
but which form the base case of inductive arguments in the theory.

	I agree with Newell and Simon that AI should try to make programs
that understand in exactly the same sense as humans do, but I am skeptical
about the specifics.

	%3It would be extremely helpful if Searle could put some
of his intuitions about the inability of machines to understand
into a form that challenges the possibility of a specific performance.
Is this possible?  I found John Haugeland's challenges to AI very
helpful in developing technical problems%1.

19  I agree that this is a good question to ask but take the other
side.

	After all the fulminations of the remainder of the paper, I am
still uncertain whether Searle thinks a computer program for the PDP-60
written by (say) Roger Schank IV could in principle leave John Searle III
undecided whether a man or a machine was at the other end of the teletype
line, giving John Searle III repeated tries and permission to consult with
experts.  I am also undecided as to whether success in this would count as
true intelligence.